Apache Kafka Development

Apache Kafka Development

• start the course
• describe the high-level consumer API for reading from Apache Kafka
• describe the simple consumer API for reading from Apache Kafka
• describe the Hadoop consumer API for reading from Apache Kafka
• configure Apache Kafka brokers
• configure Apache Kafka consumers
• configure Apache Kafka producers
• configure compression in Apache Kafka
• describe the producer API in Apache Kafka
• describe the SyncProducer API in Apache Kafka
• describe the AsyncProducer API in Apache Kafka
• configure message acknowledgement, or acking, in Apache Kafka
• batch messages in Apache Kafka
• specify keyed and non-keyed messages in Apache Kafka
• configure broker discovery in Apache Kafka
• use Apache Kafka test suites for testing
• configure serialization and deserialization in Apache Kafka
• build a custom serializer in Apache Kafka
• configure a broker and create a producer and a consumer in Apache Kafka

Apache Kafka Operations

Apache Kafka Operations

• start the course
• describe the function of Apache Kafka
• describe the architecture of Apache Kafka
• describe Apache Kafka topics
• describe Apache Kafka partitions
• describe Apache Kafka replicas
• describe Apache Kafka producers
• describe Apache Kafka consumers
• describe Apache Kafka brokers
• describe common hardware and OS specifications and their impact in Apache Kafka
• describe the main options to deploy Apache Kafka
• deploy Apache Kafka to Red Hat and CentOS
• deploy Apache Kafka to Puppet
• add and remove a broker in Apache Kafka
• move data and partitions in Apache Kafka for performance purposes
• add a new topic in Apache Kafka
• scale a producer in Apache Kafka
• scale a consumer in Apache Kafka
• monitor Apache Kafka using the web console
• monitor Apache Kafka using the offset monitor
• monitor Apache Kafka using Graphite
• monitor Apache Kafka using JMX
• monitor Apache Kafka using the log files
• tune the Linux kernel for Apache Kafka
• tune Linux systems disk throughput for Apache Kafka
• tune the Java VM for Apache Kafka
• configure and manage Apache Kafka

Clustering

Clustering

• start the course
• describe the availability and durability guarantees of Apache Kafka
• set up and configure a multi-broker cluster
• configure and balance logs and leadership for replication
• describe how log compaction works
• configure and use the log cleaner
• increase the replication factor for a topic
• add servers to a Kafka cluster and migrate data to the new servers
• mirror data between clusters such as between two data centers
• balance replicas across zones or racks |w to prevent rack-failure data loss
• set preferred replicas for leadership and handle unclean leader elections

Kafka Integration with Spark

Kafka Integration with Spark

• start the course
• install and configure the Spark Streaming package for Kafka
• read data into Spark from Kafka
• read data in parallel into Spark from Kafka
• write data back to Kafka from Spark
• write data back to Kafka from Spark in parallel
• create a direct stream to access Kafka data from Spark
• use LocationStrategies and ConsumerStrategies to improve performance
• use an RDD in cases where batch processing would be a better solution
• use offsets to handle exactly-once semantics
• use Kafka and Spark to split words from sentences

Kafka Integration with Storm

Kafka Integration with Storm

• start the course
• install and configure Storm and Kafka
• describe the Storm-Kafka pipeline for processing events
• configure the Kafka Spout for adding stream data into Storm
• read data from Kafka into Storm
• write data back to Kafka from a Storm Bolt
• create a simple Kafka-Storm wordcount application
• use the Kafka-Storm-Starter to bootstrap creating Kafka-Storm applications
• describe how Kafka can be integrated with Spark and Storm

Real-time Applications

Real-time Applications

• start the course
• describe the real-time capabilities of Kafka
• install and set up the Twitter Streaming API and framework to create a real-time Twitter application
• create a Kafka Producer to process tweets
• create a Consumer to transfer tweets to Spark
• use Spark to process tweets and take action on specific words
• create a Consumer to transfer tweets to Storm
• use Storm to process the words in incoming tweets
• use Kafka, Spark, and Storm in a pipeline to process words in a sentence
• create a simple real-time application using Kafka, Spark, and Storm